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REAL PARTY IN INTEREST 

The real party in interest in this appeal is the following party: International Business 
Machines Corporation of Armonk, New York. 
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RELATED APPEALS AND INTERFERENCES 

This appeal has no related proceedings or interferences. 
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STATUS OF CLAIMS 



TOTAL NUMBER OF CLAIMS IN APPLICATION 

The claims in the application are: 1-51 

STATUS OF ALL THE CLAIMS IN APPLICATION 

Claims canceled: 6, 8-11, 14, 16-20, 23-50 

Claims withdrawn from consideration but not canceled: None 

Claims pending: 1-5, 7, 12, 13, 15, 21, 22 and 51 

Claims allowed: None 

Claims rejected: 1-5, 7, 12, 13, 15, 21, 22 and 51 
Claims objected to: None 

CLAIMS ON APPEAL 

The claims on appeal are: 1-5, 7, 12, 13, 15, 21, 22 and 51 
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STATUS OF AMENDMENTS 



No amendment was filed after final rejection. 
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SUMMARY OF CLAIMED SUBJECT MATTER 



The present invention provides a test apparatus, system, and method by which a student's 
testing progress can be monitored from a remote location in order to make sure that the student does 
not spend an inordinate about of time on any one question during an examination. Moreover, the 
apparatus, system, and method provide for proctoring of an on-line test by a third party that is 
capable of proctoring the exam from a remote location, and providing time alert notification to the 
student during the examination in an effort to aid the student in being able to complete the exam in 
the allotted test taking period. 

A. CLAIM 1 - INDEPENDENT 

The subject matter of claim 1 is directed to a computer-implemented method for monitoring 
responses to test questions presented in a data processing system {Specification page 13, line 28 - 
page 14, line 21; page 19, lines 31-33; Figure 4, element 400}. The method comprising the 
computer implemented steps of: identifying presentation of the test questions on the data processing 
system {Specification page 20, lines 26-32; page 30, lines 16-20; Figure 10, element 1016}; 
responsive to the presentation of the test questions on the data processing system, monitoring test 
question timing data in which the test question timing data represents an elapsed time since an 
answered question from the test questions has been presented, wherein the elapsed time is an 
amount of time in attempting to answer a specific test question {Specification page 22, lines 17- 
21}; and generating a customized user alert after the test question timing data exceeds an alert 
threshold while continuing to present the specific test question for the test taker to answer 
{Specification page 22, lines 21-30}, where the customized user alert apprises a test taker that the 
elapsed time is excessive for the specific test question {Specification page 25, lines 20-27}. The 
customized user alert is periodically presented to the test taker based on an alert schedule for the 
specific test question after at least one time period indicated by the alert schedule {Specification 
page 30, line 26 - page 31, line 9; Figure 6, elements 630 and 632; Figure 10, elements 1020 
and 1028}. The customized user alert includes an amount of time remaining to answer the specific 
test question {Specification page 22, lines 13-15}, a suggested amount of time to spend on the 
specific test question { Specification page 14, lines 4-9}, and an amount of time allotted to 
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complete all of the test questions {Specification page 13, line 29 - page 14, line 9}. The alert 
schedule is generated for the test taker based on a customized alert profile for the test taker 
{Specification page 29, lines 10-14; Figure 9, element 920}. The suggested amount of time to 
spend on the specific test question is calculated for the test taker based on the customized alert 
profile {Specification page 14, lines 4-9}. The customized alert profile includes previous 
performance information of the test taker {Specification page 27, lines 27-28; Figure 9, element 
910}, information to associate a level of difficulty of the specific test question with a capability 
category of the test taker to answer the specific test question {Specification page 24, lines 2-8}, 
and at least one alert threshold for each of the test questions { Specification page 25, lines 20-27}. 

B. CLAIM 51 - INDEPENDENT 

The subject matter of claim 51 is directed to a computer-implemented method for 
monitoring responses to test questions presented in a data processing system {Specification page 
13, line 28 - page 14, line 21; page 19, lines 31-33; Figure 4, element 400}. The method 
comprising the steps of: administering, from an examination server, a plurality of tests to a plurality 
of remotely located users on a plurality of user devices {Specification page 8, lines 9-19; Figure 1, 
elements 104, 108, 111 and 114} and for each test of said plurality of tests that is administered: 
establishing a session identification for the administration of the test to the remotely located user, 
where the session identification includes a user identification, a test identifier, and a proctor device 
identifier {Specification page 15, line 25 - page 16, line 14; Figure 4, element 440}; identifying 
presentation of the test questions on a user device of said user devices {Specification page 20, 
lines 26-32; page 30, lines 16-20; Figure 10, element 1016}; responsive to the presentation of the 
test questions on said user device, monitoring test question timing data in which the test question 
timing data represents an elapsed time since an answered question from the test questions has been 
presented, wherein the elapsed time is an amount of time in attempting to answer a specific test 
question {Specification page 22, lines 17-21}; correlating the test question timing data to the 
administration of the test to the remotely located user based on the session identification; where the 
test question timing data is output to said proctor device, based on said proctor device identifier 
{ Specification page 16, lines 15-22}, in response to determining that evidence of greater than 
expected response time to the specific test question is present {Specification page 22, lines 21-30}; 
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and generating a customized user alert message after the test question timing data exceeds an alert 
threshold while continuing to present the specific test question for the remotely located user to 
answer {Specification page 22, lines 21-30}, where the customized user alert message apprises the 
remotely located user that the elapsed time is excessive for the specific test question {Specification 
page 25, lines 20-27}. The customized user alert message is periodically presented to the remotely 
located user based on an alert schedule for the specific test question after at least one time period 
indicated by the alert schedule {Specification page 30, line 26 - page 31, line 9; Figure 6, 
elements 630 and 632; Figure 10, elements 1020 and 1028}. The customized user alert message 
includes an amount of time remaining to answer the specific test question { Specification page 22, 
lines 13-15}, a suggested amount of time to spend on the specific test question {Specification 
page 14, lines 4-9}, and an amount of time allotted to complete all of the test questions 
{Specification page 13, line 29 - page 14, line 9}. The alert schedule is generated for the remotely 
located user based on a customized alert profile for the remotely located user {Specification page 
29, lines 10-14; Figure 9, element 920}. The suggested amount of time to spend on the specific 
test question is calculated for the remotely located user based on the customized alert profile 
{Specification page 14, lines 4-9}. The customized alert profile includes previous performance 
information of the remotely located user {Specification page 27, lines 27-28; Figure 9, element 
910}, information to associate a level of difficulty of the specific test question with a capability 
category of the remotely located user to answer the specific test question {Specification page 24, 
lines 2-8}, and a plurality of alert thresholds for each of the test questions { Specification page 25, 
lines 20-27}. The remotely located user can send instant messages to and receive instant messages 
from said proctor device associated with said examination server and wherein said proctor device 
can send instant messages to and receive instant messages from the plurality of remotely located 
users, and wherein instant messages are used to communicate and clarify test question wording 
details, test instructions, and the test question timing data for the remotely located user during the 
test {Specification page 21, lines 23-31; Figure 5, element 520}. 
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GROUNDS OF REJECTION TO BE REVIEWED ON APPEAL 



The grounds of rejection to review on appeal are as follows: 

A. GROUND OF REJECTION 1 

Rejection of Claims 1-3, 12, 21 and 22 under 35 U.S.C. § 103 as being unpatentable over 
Thomas (U.S. Patent No. 5,885,087); 

B. GROUND OF REJECTION 2 

Rejection of Claims 4, 5, 7 and 13 under 35 U.S.C. § 103 as being unpatentable over 
Thomas (U.S. Patent No. 5,885,087) in view of Sugimoto (U.S. Patent No. 6,755,661) and further 
in view of admitted prior art; 

C. GROUND OF REJECTION 3 

Rejection of Claim 15 under 35 U.S.C. § 103 as being unpatentable over Thomas (U.S. 
Patent No. 5,885,087) in view of Sugimoto (U.S. Patent No. 6,755,661) and further in view 
Kershaw et al. (U.S. Patent No. 5,827,070); and 

D. GROUND OF REJECTION 4 

Rejection of Claim 51 under 35 U.S.C. § 103 as being unpatentable over Kershaw in view 
of Thomas (U.S. Patent No. 5,885,087), Hoehn-Saroc (U.S. Patent No. 5,915,973) and Sugimoto 
(U.S. Patent No. 6,755,661). 



(Appeal Brief Page 9 of 24) 
Banerjee et al. - 09/941,251 



ARGUMENT 



The present invention provides a test apparatus, system, and method by which a student's 
testing progress can be monitored from a remote location in order to make sure that the student does 
not spend an inordinate about of time on any one question during an examination. Moreover, the 
apparatus, system, and method provide for proctoring of an on-line test by a third party that is 
capable of proctoring the exam from a remote location, and providing time alert notification to the 
student during the examination in an effort to aid the student in being able to complete the exam in 
the allotted test taking period. 

A. GROUND OF REJECTION 1 (Claims 1-3, 12, 21 and 22) 

Claims 1-3, 12, 21 and 22 stand rejected under 35 U.S.C. § 103 as being unpatentable over 
Thomas (U.S. Patent No. 5,885,087), hereinafter "Thomas". 

1. Claims 1, 2, 12, 21 and 22 

With respect to Claim 1, such claim recites "wherein the customized user alert is 
periodically presented to the test taker based on an alert schedule for the specific test question after 
at least one time period indicated by the alert schedule, wherein the customized user alert includes 
an amount of time remaining to answer the specific test question, a suggested amount of time to 
spend on the specific test question, and an amount of time allotted to complete all of the test 
questions". As can be seen, there is a customized user alert that is periodically presented to the user 
based on an alert schedule for the specific test question, and this customized user alert is presented 
to the test taker after at least one time period indicated by the alert schedule. In addition, per these 
features of Claim 1, this same customized user alert includes (1) an amount of time remaining to 
answer the specific test question , (2) a suggested amount of time to spend on this specific test 
question , and (3) an amount of time allotted to complete all of the test questions. None of the cited 
references teach or suggest such a customized user alert that includes all of these claimed features - 
nor has the Examiner alleged a teaching/suggestion of such a customized user alert having all of 
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these claimed features. Therefore, Claim 1 has been erroneously rejected 1 as the Examiner has 
failed to properly establish a prima facie showing of obviousness with respect to the claimed 
customized user alert that includes times pertaining to both the overall test as well as the particular 
question. 

Specifically with respect to the claimed customized user alert, the Examiner merely alleges 
that the Thomas describes that the time taken by a user to answer a question is (i) tracked, (ii) 
displayed, and (iii) compared to a predetermined time. This assertion does not establish any 
teaching or suggestion of the claimed customized user alert. In addressing the cited Sugimoto 
reference, the Examiner alleges that Sugimoto describes 'a preset time limit for each question'. 
These are the only time-based teachings that are alleged to be taught by the cited references. Thus, 
the Examiner has failed to establish a teaching or suggestion of the specific time-based features 
pertaining to the customized user alert, including that the customized user alert is periodically 
presented to the test taker based on an alert schedule for the specific test question after at least one 
time period indicated by the alert schedule, wherein the customized user alert includes (1) an 
amount of time remaining to answer the specific test question , (2) a suggested amount of time to 
spend on the specific test question , and (3) an amount of time allotted to complete all of the test 
questions. Thus, it is urged that Claim 1 has been erroneously rejected due to this prima facie 
obviousness deficiency with respect to the three time parameters associated with the customized 
user alert that is periodically sent to the user. 

In addition, Claim 1 includes detailed characteristics associated with a customized alert 
profile (as contrasted with the customized user alert previously described hereinabove). Claim 1 
also recites "wherein the alert schedule is generated for the test taker based on a customized alert 

1 If the examiner fails to establish a prima facie case, the rejection is improper and will be overturned. 
In re Fine, 837 F.2d 1071, 1074, 5 USPQ2d 1596, 1598 (Fed. Cir. 1988). In the absence of a proper 
prima facie case of obviousness, an applicant who complies with the other statutory requirements is 
entitled to a patent. See In re Oetiker, 977 F.2d 1443, 1445, 24 USPQ2d 1443, 1444 (Fed. Cir. 1992). 

2 In rejecting claims under 35 U.S.C. Section 103, the examiner bears the initial burden of presenting a 
prima facie case of obviousness. In re Oetiker, 977 F.2d 1443, 1445, 24 USPQ2d 1443, 1444 (Fed. Cir. 
1992). Only if that burden is met, does the burden of coming forward with evidence or argument shift to 
the applicant. Id. All words in a claim must be considered in judging the patentability of that claim 
against the prior art." MPEP 2143.03; In re Wilson, 424 F.2d 1382, 1385, 165 USPQ 494, 496 (CCPA 
1970) (emphasis added by Applicants). 
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profile for the test taker, wherein the suggested amount of time to spend on the specific test 
question is calculated for the test taker based on the customized alert profile, and wherein the 
customized alert profile includes previous performance information of the test taker, information to 
associate a level of difficulty of the specific test question with a capability category of the test taker 
to answer the specific test question, and at least one alert threshold for each of the test questions". 
As can be seen, this customized alert profile (that is used in generating the alert schedule that is 
used to send the customized user alerts that were previously described) includes (1) previous 
performance information of the test taker, (2) information to associate a level of difficulty of the 
specific test question with a capability category of the test taker to answer the specific test question, 
and (3) at least one alert threshold for each of the test questions. None of the cited references teach 
or suggest such a customized user profile. 

In rejecting this aspect of Claim 1, the Examiner states that Thomas describes player 
profiles in order to provide a history of the user's progress including performance by subject or 
topic. At a fundamental level, such Thomas player profile is not used in generating any type of alert 
schedule (that is used to present a customized user alert to a test taker), as is provided by the 
features of Claim 1. Further, such Thomas player profile does not include: (i) information to 
associate a level of difficulty of the specific test question with a capability category of the test taker 
to answer the specific test question, or (ii) at least one alert threshold for each of the test questions, 
as is provided by the features of Claim 1 . 

As to the teachings of Sugimoto and a profile, the Examiner merely alleges that Sugimoto 
describes test taker profiles that include 'skill level of the user', and that the profile is used to 
determine questions that are presented to the test taker. At a fundamental level, such Sugimoto 
profile is not used in generating any type of alert schedule (that is used to present a customized user 
alert to a test taker), as is provided by the features of Claim 1. Further, such Sugimoto profile does 
not include: (i) information to associate a level of difficulty of the specific test question with a 
capability category of the test taker to answer the specific test question, or (ii) at least one alert 
threshold for each of the test questions, as is provided by the features of Claim 1. 

Thus, as neither cited reference describes (1) a user profile that includes: (i) information to 
associate a level of difficulty of the specific test question with a capability category of the test taker 
to answer the specific test question, or (ii) at least one alert threshold for each of the test questions; 
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or (2) a user profile that is used to generate an alert schedule (and this alert schedule is used to 
periodically present a customized user alert to a test taker, as previously described), it is further 
urged that Claim 1 has been erroneously rejected due to this prima facie obviousness deficiency 
with respect to the characteristics and use of the claimed customized alert profile. 

In conclusion with respect to Claim 1, none of the cited references teach or suggest (1) the 
three different time parameters associated with the customized user alert that is periodically sent to 
the user (which advantageously facilitates providing the test taker with detailed time parameter 
information pertaining to various aspects of taking the test), (2) a user profile that includes: (i) 
information to associate a level of difficulty of the specific test question with a capability category 
of the test taker to answer the specific test question, or (ii) at least one alert threshold for each of the 
test questions; or (3) a user profile that is used to generate an alert schedule (and this alert schedule 
is advantageously used to periodically present a customized user alert to a test taker). Therefore, it 
is urged that Claim 1 has been erroneously rejected, and further that such claim is in condition for 
allowance. 

2. Claim 3 

Applicants initially urge error in the rejection of Claim 3 for reasons given above with 
respect to Claim 1 (of which Claim 3 depends upon). 

Further with respect to Claim 3, such claim recites "wherein the presentation of the test 
questions is based on levels of difficulty of the test questions and the capability category of the test 
taker to answer the test questions". As can be seen, the presentation of the test questions is based 
on two criteria: (1) levels of difficulty of the test questions, and (2) the capability category of the 
test taker to answer the test questions. None of the cited references teach or suggest these two 
criteria, in combination, being used in the presentation of test questions. For example, Sugimoto 
describes a question database that stores questions, and this question database includes the number 
of solvers and the number of solvers who return the correct answer (Sugimoto col. 6, lines 3-5), and 
a skill category code (Sugimoto col. 6, line 6). Neither of these stored database items describe that 
the presentation of the test questions is based on levels of difficulty of the test questions and the 
capability category of the test taker to answer the test questions. Thus, it is further shown that 
Claim 3 has been erroneously rejected due to this additional prima facie obviousness deficiency. 
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B. GROUND OF REJECTION 2 (Claims 4, 5, 7 and 13) 

Claims 4, 5, 7 and 13 stand rejected under 35 U.S.C. § 103 as being unpatentable over 
Thomas in view of Sugimoto (U.S. Patent No. 6,755,661), hereinafter "Sugimoto" and further in 
view of admitted prior art. 

1. Claims 4, 5, 7 and 13 

Applicants urge error in the rejection of Claims 4, 5, 7 and 13 for similar reasons to those 
given above with respect to Claim 1 (of which Claims 4, 5, 7 and 13 depend upon), as the newly 
cited reference to Sugimoto and the alleged admitted prior art do not overcome the 
teaching/suggestion deficiencies identified hereinabove with respect to such claims. 

Therefore, the rejection of Claims 4, 5, 7 and 13 under 35 U.S.C. § 103 has been overcome. 

C. GROUND OF REJECTION 3 (Claim 15) 

Claim 15 stands rejected under 35 U.S.C. § 103 as being unpatentable over Thomas in view 
of Sugimoto and further in view Kershaw et al. (U.S. Patent No. 5,827,070), hereinafter "Kershaw". 

1. Claim 15 

Applicants initially urge error in the rejection of Claim 15 for similar reasons to those given 
above with respect to Claim 1 (of which Claim 15 depends upon), as the newly cited reference to 
Kershaw does not overcome the teaching/suggestion deficiencies identified hereinabove. 

Applicants further urge error in the rejection of Claim 15 by showing that none of the cited 
references teach or suggest the claimed feature of "wherein a session identification is established 
for the presentation of the test questions by a proctor device and includes a proctor device identifier, 
and wherein outputting the test question timing data to the proctor device is based on the proctor 
device identifier". Specifically, none of the cited references teach or suggest (1) a proctor device 
that establishes a session identification for presentation of test questions, or (2) the outputting of test 
question timing data to such proctor device based on a proctor device identifier. In rejecting Claim 
15, the Examiner merely alleges that Kershaw (i) discloses the collection of statistical data on all 
test takers (which does not establish a teaching of the claimed proctor device characteristics (1) and 
(2) identified above), and (ii) recording of a test program id, registration id, test center id, and 
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workstation id for each tester (which does not establish a teaching of the claimed proctor device 
characteristics (1) and (2) identified above). Thus, it is further shown that Claim 15 has been 
erroneously rejected due to this additional prima facie obviousness deficiency with respect to the 
proctor device and its associated characteristics. 

D. GROUND OF REJECTION 4 (Claim 51) 

Claim 51 stands rejected under 35 U.S.C. § 103 as being unpatentable over Kershaw in 
view of Thomas (U.S. Patent No. 5,885,087), Hoehn-Saroc (U.S. Patent No. 5,915,973) and 
Sugimoto (U.S. Patent No. 6,755,661). 

1. Claim 51 

With respect to Claim 51, such claim recites "wherein the test question timing data is output 
to said proctor device, based on said proctor device identifier, in response to determining that 
evidence of greater than expected response time to the specific test question is present" and 
"wherein the remotely located user can send instant messages to and receive instant messages from 
said proctor device associated with said examination server and wherein said proctor device can 
send instant messages to and receive instant messages from the plurality of remotely located users". 
None of the cited references teach or suggest - nor has the Examiner alleged any teaching or 
suggestion of - such test question timing data being output to a proctor device, where such proctor 
device facilitates instant messaging communication with a plurality of remotely located users. 
Thus, Claim 51 has been erroneously rejected as the Examiner has failed to establish a prima facie 
showing of obviousness with respect to Claim 51. 

Further with respect to Claim 51, such claim recites "generating a customized user alert 
message after the test question timing data exceeds an alert threshold while continuing to present 
the specific test question for the remotely located user to answer, wherein the customized user alert 
message apprises the remotely located user that the elapsed time is excessive for the specific test 
question, wherein the customized user alert message is periodically presented to the remotely 
located user based on an alert schedule for the specific test question after at least one time period 
indicated by the alert schedule, wherein the customized user alert message includes an amount of 
time remaining to answer the specific test question, a suggested amount of time to spend on the 
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specific test question, and an amount of time allotted to complete all of the test questions". As can 
be seen, there is a customized user alert that is generated after the test question timing data exceeds 
an alert threshold while continuing to present the specific test question for the remotely located 
user to answer. In addition, per these features of Claim 51, this same customized user alert 
message (1) apprises the remotely located user that the elapsed time is excessive for the specific test 
question, (2) is periodically presented to the remotely located user based on an alert schedule for the 
specific test question after at least one time period indicated by the alert schedule, and (3) includes 
(i) an amount of time remaining to answer the specific test question, (ii) a suggested amount of time 
to spend on the specific test question, and (iii) an amount of time allotted to complete all of the test 
questions. None of the cited references teach or suggest such a customized user alert that includes 
all of these claimed features - nor has the Examiner alleged a teaching/suggestion of such a 
customized user alert message having all of these claimed features. Therefore, Claim 51 has been 
erroneously rejected as the Examiner has failed to properly establish a prima facie showing of 
obviousness. 

In addition, Claim 51 includes details characteristics associated with a customized alert 
profile (as contrasted with the customized user alert previously described hereinabove). Claim 51 
also recites "wherein the alert schedule is generated for the remotely located user based on a 
customized alert profile for the remotely located user, wherein the suggested amount of time to 
spend on the specific test question is calculated for the remotely located user based on the 
customized alert profile, and wherein the customized alert profile includes previous performance 
information of the remotely located user, information to associate a level of difficulty of the specific 
test question with a capability category of the remotely located user to answer the specific test 
question, and a plurality of alert thresholds for each of the test questions". As can be seen, this 
customized alert profile (that is used in generating the alert schedule that is used to present the 
customized user alert messages to a user, as previously described) includes (1) previous 
performance information of the test taker, (2) information to associate a level of difficulty of the 
specific test question with a capability category of the test taker to answer the specific test question, 
and (3) a plurality of alert thresholds for each of the test questions. None of the cited references 
teach or suggest such a customized alert profile. 
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In rejecting this aspect of Claim 51, the Examiner alleges that Sugimoto describes test taker 
profiles that include 'skill level of the user', and that the profile is used to determine questions that 
are presented to the test taker. At a fundamental level, such Sugimoto profile is not used in 
generating any type of alert schedule (that is used to present the customized user alert messages to a 
user, as previously described), as is provided by the features of Claim 51. Further, such Sugimoto 
profile does not include: (i) information to associate a level of difficulty of the specific test question 
with a capability category of the test taker to answer the specific test question, or (ii) at least one 
alert threshold for each of the test questions, as is provided by the features of Claim 1. 

Thus, as none of the cited references describes (1) timing data being output to a proctor 
device, where such proctor device facilitates instant messaging communication with a plurality of 
remotely located users, (2) a user profile that includes: (i) information to associate a level of 
difficulty of the specific test question with a capability category of the test taker to answer the 
specific test question, or (ii) at least one alert threshold for each of the test questions; or (3) a user 
profile that is used to generate an alert schedule (and this alert schedule is used to periodically 
present a customized user alert to a test taker, as previously described), it is urged that Claim 51 has 
been erroneously rejected due to this prima facie obviousness deficiency. 

Therefore, the rejection of Claim 51 under 35 U.S.C. § 103 has been overcome. 

E. CONCLUSION 

As shown above, the Examiner has failed to state valid rejections against any of the claims. 
Therefore, Appellants request that the Board of Patent Appeals and Interferences reverse the 
rejections. Additionally, Appellants request that the Board direct the Examiner to allow the claims. 

Date: January 30, 2009 Respectfully submitted, 

/Wayne P. Bailey/ 
Wayne P. Bailey 
Reg. No. 34,289 
Yee & Associates, P.C. 
P.O. Box 802333 
Dallas, TX 75380 
(972) 385-8777 
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CLAIMS APPENDIX 



The text of the claims involved in the appeal is as follows: 

1 . A computer- implemented method for monitoring responses to test questions presented in 
a data processing system, the method comprising the computer implemented steps of: 

identifying presentation of the test questions on the data processing system; 

responsive to the presentation of the test questions on the data processing system, 
monitoring test question timing data in which the test question timing data represents an elapsed 
time since an answered question from the test questions has been presented, wherein the elapsed 
time is an amount of time in attempting to answer a specific test question; and 

generating a customized user alert after the test question timing data exceeds an alert 
threshold while continuing to present the specific test question for the test taker to answer, 
wherein the customized user alert apprises a test taker that the elapsed time is excessive for the 
specific test question, wherein the customized user alert is periodically presented to the test taker 
based on an alert schedule for the specific test question after at least one time period indicated by 
the alert schedule, wherein the customized user alert includes an amount of time remaining to 
answer the specific test question, a suggested amount of time to spend on the specific test 
question, and an amount of time allotted to complete all of the test questions, wherein the alert 
schedule is generated for the test taker based on a customized alert profile for the test taker, 
wherein the suggested amount of time to spend on the specific test question is calculated for the 
test taker based on the customized alert profile, and wherein the customized alert profile includes 
previous performance information of the test taker, information to associate a level of difficulty 
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of the specific test question with a capability category of the test taker to answer the specific test 
question, and at least one alert threshold for each of the test questions. 

2. The computer- implemented method of claim 1, wherein the monitoring step is performed 
by a proctor device or a program on the data processing system. 

3. The computer- implemented method of claim 1, wherein the presentation of the test 
questions is based on levels of difficulty of the test questions and the capability category of the 
test taker to answer the test questions. 

4. The computer-implemented method of claim 1 further comprising: 
billing a client for monitoring the presentation of the test questions. 

5. The computer- implemented method of claim 1, wherein the test questions are part of a 
test and further comprising: 

storing an identification of a number of test takers for the test; and 
billing a client based on the number of test takers for the test. 

7. The computer- implemented method of claim 1, further comprising: 

billing a test developer for monitoring the presentation of the test questions. 

12. The computer- implemented method of claim 1, further comprising! 
storing a score for the test questions in a permanent storage. 
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13. The computer-implemented method of claim 1, wherein the test questions are developed 
by a test developer system and wherein the method is implemented by a test administration 
system that is operated by a different entity from the test developer system. 

15. The computer- implemented method of claim 1, wherein a session identification is 
established for the presentation of the test questions by a proctor device and includes a proctor 
device identifier, and wherein outputting the test question timing data to the proctor device is 
based on the proctor device identifier. 

21. The computer- implemented method of claim 1, further comprising: 

storing a response to the specific test question from the test taker to update the 
customized alert profile for use in future tests. 

22. The computer- implemented method of claim 1, further comprising: 

storing the test question timing data for the specific test question to update the 
customized alert profile for use in future tests. 

51. A computer-implemented method for monitoring responses to test questions presented in 
a data processing system, the method comprising the steps of: 

administering, from an examination server, a plurality of tests to a plurality of 
remotely located users on a plurality of user devices and for each test of said plurality of 
tests that is administered: 

establishing a session identification for the administration of the test to the 
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remotely located user, wherein said session identification includes a user 

identification, a test identifier, and a proctor device identifier; 

identifying presentation of the test questions on a user device of said user devices; 

responsive to the presentation of the test questions on said user device, 
monitoring test question timing data in which the test question timing data 
represents an elapsed time since an answered question from the test questions has 
been presented, wherein the elapsed time is an amount of time in attempting to 
answer a specific test question; 

correlating the test question timing data to the administration of the test to 
the remotely located user based on the session identification; 

wherein the test question timing data is output to said proctor device, 
based on said proctor device identifier, in response to determining that evidence 
of greater than expected response time to the specific test question is present; and 

generating a customized user alert message after the test question timing 
data exceeds an alert threshold while continuing to present the specific test 
question for the remotely located user to answer, wherein the customized user 
alert message apprises the remotely located user that the elapsed time is excessive 
for the specific test question, wherein the customized user alert message is 
periodically presented to the remotely located user based on an alert schedule for 
the specific test question after at least one time period indicated by the alert 
schedule, wherein the customized user alert message includes an amount of time 
remaining to answer the specific test question, a suggested amount of time to 
spend on the specific test question, and an amount of time allotted to complete all 
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of the test questions, wherein the alert schedule is generated for the remotely 
located user based on a customized alert profile for the remotely located user, 
wherein the suggested amount of time to spend on the specific test question is 
calculated for the remotely located user based on the customized alert profile, and 
wherein the customized alert profile includes previous performance information 
of the remotely located user, information to associate a level of difficulty of the 
specific test question with a capability category of the remotely located user to 
answer the specific test question, and a plurality of alert thresholds for each of the 
test questions; 

wherein the remotely located user can send instant messages to and receive 
instant messages from said proctor device associated with said examination server 
and wherein said proctor device can send instant messages to and receive instant 
messages from the plurality of remotely located users, and wherein instant 
messages are used to communicate and clarify test question wording details, test 
instructions, and the test question timing data for the remotely located user during 
the test. 
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EVIDENCE APPENDIX 



This appeal brief presents no additional evidence. 
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RELATED PROCEEDINGS APPENDIX 

This appeal has no related proceedings. 
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